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ABSTRACT 



To raise vocational completers' academic achievement through 
school reform, the High Schools That Work (HSTW) network of the Southern 
Regional Education Board (SREB) promotes 10 key practices for raising 
academic achievement . The impact of these key practices was examined in a 
study of how 424 schools adopted them in the following six clusters : 
curriculum standards; instructional goals; academic/vocational integration; 
guidance counseling; teacher practices; and work-based learning. The study 
used data from multiple sources, including teacher and student surveys, high 
school transcripts, and achievement test scores. Overall, increases in the 
proportion of students meeting the HSTW curriculum standards had a large 
impact on achievement gains in science, reading, and mathematics. Changes in 
the proportion of students perceiving that their academic and vocational 
teachers were working together to improve students' academic skills had 
almost as much positive effect as curriculum changes did. Increases in the 
amount of time students spent talking to their guidance counselors and 
teachers about their school program were directly associated with increases 
in the schools' mean assessment scores. The other clusters seemed to have 
little or no explanatory power for predicting school changes in academic 
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achievement. (Twenty- two tables/figures are included. Appended are the 10 
HSTW key practices . ) (MN) 
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EXECUTIVE SUMMARY 

School-wide efforts to improve the education of American students have been 
implemented in many schools throughout the nation. The Southern Regional Education 
Board’s High Schools That Work (HSTW) network stands out as one of the few consortia 
to coordinate that effort and to collect and analyze data as part of a service to its participants. 
On a biannual basis, the participating schools assess their graduating vocational completers 
in science, mathematics, and reading using the HSTW Assessment. They also collect data 
on student course-taking patterns, student behaviors and attitudes, and teacher attitudes and 
characteristics. In addition to creating useful comparison data for benchmarking the 
progress of individual sites, the assessments enable SREB to test theories about basic 
associations between certain practices or attitudes and student outcome measures. 

Several underlying questions, however, remained unanswered, such as, “Can we 
look into the ‘black box’ of whole school reform and provide evidence of particularly 
effective practices?” Using the test scores from 1996 and 1998, demographic variables to 
control for changes in the tested student body, and variables that correspond to the key 
practices of High Schools That Work , this analytic study attempts to provide insight 
regarding individual practices or program elements. In order to reach findings that might 
prove useful to schools attempting to raise student achievement, all data were aggregated to 
the school level. 

For the 424 schools in this study, the mean gain in the three assessment subjects 
between 1996 and 1998 ranged from 4 to 13 points. We looked specifically at six 
clusters to represent the key practices promoted by HSTW: (1) curriculum standards, 
(2) instructional goals, (3) academic/vocational integration, (4) guidance counseling, 
(5) teacher practices, and (6) work-based learning. Some of the clusters were more easily 
captured by data elements than were others. In addition, it appears that some clusters were 
more operational within schools than were others. In other words, schools had room for 
improvement and made positive changes between 1996 and 1998 for some clusters, while 
for others, the opportunity for improvement on these measures was slight or not taken 
advantage of. 

This analysis predominantly explores the individual impact of each cluster on 
student achievement, while controlling for changes in student demographics. Overall, 
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increases in the proportion of students meeting HSTW curriculum standards had a large 
impact on achievement gains in science, reading, and math. Changes in the proportion of 
students perceiving that their academic and vocational teachers were working together to 
improve students’ mathematics, reading, and writing skills had almost as much positive 
effect in the statistical model as curriculum changes. Likewise, increases in the amount of 
time that students spent talking to their guidance counselors and teachers about their school 
program were directly associated with increases in the schools’ mean assessment scores. 
The other clusters seemed to have little or no explanatory power for predicting school 
changes in student academic achievement. 

In any analysis of schools, cause and effect are difficult to determine, and 
corresponding data are difficult to collect. Our primary purpose in this study is to examine 
the correlates of success in the HSTW network using the HSTW Assessment and survey 
data; however, we also hope that this analysis — using fairly simple models with school- 
level data — might spark others to consider similar data presently used for report cards as a 
source for thoughtful research and study. 
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INTRODUCTION 

School-wide reform efforts have been in favor for several years among educational 
policymakers. Whether it is called “whole school reform,” “comprehensive reform,” or 
“school-wide reform,” much of the educational research community has endorsed this 
approach to improving school achievement. Reflecting the growing acceptance of this 
approach among educators, Congress passed the Comprehensive School Reform 
Demonstration Program (Obey-Porter) in 1994. This bill specifically endorsed 24 reform 
strategies that were classified as “whole school” reform approaches. While there is wide 
agreement on the theoretical benefits of whole school reform, there have been few 
empirical studies of the outcomes of these reform efforts. For example, in a review of the 
research literature on the 24 programs cited in Comprehensive School Reform 
Demonstration Program, the American Institutes for Research (AIR) found few studies of 
student outcomes that had what they considered “rigorous” research designs (American 
Institutes for Research, 1999). This study attempts to fill some of this void by looking at 
the student outcomes of one the programs included in the Obey-Porter Bill — the High 
Schools That Work ( HSTW) initiative. 

HSTW began in 1988 as a pilot project of the Southern Regional Education 
Board’s (SREB) Vocational Education Consortium with a group of 28 schools across the 
southeastern United States. In the past 10 years, more than 900 school sites have joined the 
HSTW consortium, and evidence suggests that many of these schools have increased 
student achievement. 

Since the beginning of the effort, SREB has used assessment scores, transcript 
data, and survey information to keep track of the progress being made by the schools in the 
consortium. After the Educational Testing Service (ETS) analyzed the results of the 1996 
HSTW Assessment, SREB announced that student test scores had increased from 1993 to 
1996 in all three tested subject areas and increased from 1994 to 1996 in reading and math. 
The average reading score increased from 267 (in 1993) and 264 (in 1994) to 272; 
mathematics scores from 285 (in 1993) and 281 (in 1994) to 286; and science scores from 
270 (in 1993) and 282 (in 1994) to 283. 1 



^rom 1993 to 1996, the average reading score increased by 0.19 standard deviations (SD=25.65); the 
average mathematics score increased by 6.03 standard deviations (SD=29.32); and the average 
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HSTW is based on the belief that all students — including students traditionally in 
the vocational or general track — can master rigorous academic curriculum if they are 
exposed to the right school environment. The “right” environment for HSTW is a school 
that “blends the essential content of traditional college-preparatory studies — mathematics, 
science, and language arts — with quality vocational and technical studies” (Bottoms & 
Mikos, 1995). Therefore, while calling for whole school reform, the HSTW initiative is 
particularly interested in the outcomes of students commonly placed in vocational or 
general studies. 

The focus of this study is the academic outcomes for students who completed a 
concentrated sequence of vocational coursework — a group referred to as vocational 
completers. Data that had already been collected by schools within the HSTW network for 
internal purposes is used in the study. In the past, these data have been used by schools to 
measure various aspects of the implementation of the HSTW program. They were not 
specifically collected to evaluate the effectiveness of the HSTW initiative. The use of these 
data produced enormous cost savings over collecting new data for an evaluation of HSTW. 

Furthermore, in many ways, the data collected by these schools are similar to the 
data now being collected by various states and localities for “school report cards.” 
Therefore, this research project was an opportunity to test the feasibility of using school- 
based data collections to provide quantitative information on school effectiveness. In the 
present study, using student and teacher survey, transcript, and ETS test score data, a 
statistical model has been developed to determine which practices were most closely 
associated with high student performance, as measured by student test scores in reading, 
mathematics, and science. 



science score increased by 0.36 standard deviations (SD=33.45). From 1994 to 1996, the average 
reading score increased by 0.19 standard deviations (SD=29.97), the average mathematics score 
increased by 0.17 standard deviations (SD=29.44), and the average science score increased by 0.03 
standard deviations (SD=29.6). 
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METHODS 
Data Sources 

This study used a variety of data from multiple sources, including teacher and 
student surveys, high school transcripts, and achievement test scores in science, 
mathematics, and reading. This section describes each data source, how the information 
was collected, and how the data were used in this analysis. 

HSTW Assessment Scores 

On a roughly biannual basis, staff at each HSTW site administer the HSTW 
Assessment — a series of tests based on the science, mathematics, and reading 
examinations included in the National Assessment of Educational Progress (NAEP) — to a 
selected group of students. These students are seniors and are expected to graduate with 
four Carnegie units of credit in a vocational concentration. In this study, this group of 
students is referred to as vocational completers. 

Of the approximately 650 sites that were part of the HSTW network during 1998, 
425 had data points for the two-year cycle. The test score data were used as dependent 
variables in the statistical analyses, measuring increases and decreases in student 
performance in science, mathematics, and reading from 1996 to 1998. 



Surveys 

In addition to the science, mathematics, and reading assessments, vocational 
students also responded to a battery of survey items as part of the HSTW Assessment. 
Vocational students reported what they were taught, how they were taught, what was 
expected of them, and what effort the school put forth. This data set also includes enough 
transcript information to determine the percentage of students who completed SREB’s 
recommended curriculum in science, mathematics, English, and vocational studies at each 
site in 1996. During the same semester that students are tested and surveyed, educational 



2 Tests have been administered in 1988, 1990, 1993, 1994, 1996, and 1998. In order to manage the 
growth of the initiative from 38 sites in 1990 to an expected 500 sites in 1994, testing was suspended 
in 1992; sites were added in two phases in the years 1993 and 1994. 

3 Most states do not define “vocational completer” and use HSTW ' s definition of four Carnegie units 
in a vocational concentration to select students who will be assessed; however, a few states or 
districts do have their own definitions, and in a few instances, they vary from the HSTW definition. 
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staff also complete a written survey. 4 Site staff responded to items concerning the 
integration of academic and vocational education, amount of time devoted to teaching basic 
academic skills, and staff development needed to achieve quality learning for more 
students. These student and teacher data, along with the student test scores, provided the 
basis for the quantitative analysis. 



Statistical Analysis Procedures 

In theory (and with an unlimited budget), assessing the effectiveness of a school 
reform effort should be straightforward. By either experimental design or quasi- 
experimental design, groups of schools are randomly assigned to control and experimental 
groups. If a group of schools that has implemented the reform package at tj has overall 
gains in achievement by t 2 and the control group of schools does not, then one can come to 
the tentative conclusion that the reform had the desired effect. In practice, many things can 
go wrong with an evaluation based on the best of research designs — not the least of which 
is actually implementing the design in the first place. To paraphrase Donald Campbell, 
experiments can turn into quasi -experiments, which too often then turn into “queasy- 
experiments.” 

However, in this study we do not have the luxury of even starting with an 
experimental or quasi-experimental design. As mentioned previously, for reasons of cost 
and practicality, a design that capitalized on the ongoing data collection efforts of SREB 
was used. Both the assessment data and the individual survey data had already been 
collected for HSTW. Furthermore, these data were originally designed to enable schools 
within HSTW to track the school’s own progress compared with the HSTW network and 
national comparisons; the data were not designed explicitly for overall evaluative purposes. 
In addition, all the schools in the HSTW sample are theoretically receiving the same 
treatment. That is, the same set of key practices are being implemented in each school — 
although as we shall see, with varying degrees of success. To further complicate matters, 
the assessment data are not longitudinal in that they do not track individual students over 
time but are sets of multiple cross-sectional data. Each survey year represents a new cohort 



4 Site coordinators are instructed to survey at least the English, mathematics, science, and 
vocational/technical teachers. They are encouraged, but not required, to give the survey to teachers of 
other subject areas. 
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of vocational completers. We are thus left with a one-group pretest/posttest design where 
the pretest and the posttest are conducted on different sets of students and the treatment has 
already been introduced, albeit at different levels of “dosage.” 

Even given the inherent deficiencies in the data, we felt that an analysis of these data 
would be worthwhile. A great deal of data has been collected from students, teachers, and 
other school staff over several years that can be used to give understanding as to what 
works in the cluster of practices within HSTW. Furthermore, many states are producing 
school “report cards” that are based on data much like the data collected by the HSTW 
network. If examining the HSTW data leads to insights into aspects of program 
effectiveness, practitioners may be able to use this same approach with their own data to 
give them similar insights into their own reform efforts. 

Study Design 

In many instances, researchers are merely interested in whether or not the program 
was effective. The explanatory variable then can become a single measurement of program 
implementation or a set of variables that represent aspects of the program. The size and 
statistical significance of the added explanatory power of these variables (measured by the 
R 2 added due to these variables) then becomes evidence for school improvement due to 
program implementation. This is the classic “black box” evaluation. 

In “whole school” reform, however, many things may be going on inside the black 
box. In the present case, practitioners at SREB are interested in peeking into the box to see 
what aspects of HSTW have been particularly effective (or ineffective). That is, what 
among the several key practices led to observed gains in school-wide achievement? One 
might be tempted to do this by using standard multivariate analysis and assessing the 
unique contribution of each independent variable within the set of variables representing the 
program (the Xj) on predicting differences in the school outcome variable (Y). This unique 
contribution could be assessed by examining the regression coefficient of Xj on Y (bj), the 
partial correlation of Xj with Y (prj ), or the semipartial correlation of Xj with Y (srj) 
(Darlington, 1990). 

However, in the present case where there are lots of things changing within the 
school — some due to reform, some not — the unique contribution of a particular practice is 
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not particularly interesting nor informative for policy. This is because schools, like most 
social institutions, are messy cause and effect soups. Many terms and analogies have been 
used to describe this messiness; such have been referred to as “loosely coupled systems” 
and as causal “garbage cans,” but the effect described is the same (Weick, 1969). In whole 
school reform, lots of things are going on simultaneously. A simple analogy may be of 
use: 



In classic Newtonian physics, one can predict with some accuracy the effect of 
hitting the cue ball on the eight ball when the cue ball is hit at a certain angle and with a 
certain force. In “whole school” reform efforts, there may be several “cue balls” that are 
hit — sometimes simultaneously, sometimes not. That is, various changes in practice or 
pedagogy may be happening within the school at the same time. This is the point of whole 
school reform. The elements of these practices are also dependent — improvement in one 
[practice one] will be associated with improvement in another [practice two], as though 
several cue balls are tied together with varying lengths of cord and are then shot at the eight 
ball. 



In the present case, SREB has tried to implement numerous practices that are 
closely related to one another — for example, setting high expectations and replacing the 
general track with a solid academic core curriculum. Therefore, this analysis looks at the 
association of each independent variable with school improvement after introducing a 
limited set of control variables rather than at the unique contribution of each independent 
variable. We explore the association of each key practice with gains in achievement by 
examining the beta coefficient for each individual variable within the set of variables 
representing the key practice. This regression coefficient results when only that variable is 
entered into the equation with the control variables — that is, the regression coefficient for 
that variable in a model that contains the control variables and only that one variable. 

We also present and discuss the amount of variance explained by the set of 
variables representing a key practice — the R 2 added. However, we would like to caution the 
reader in over-interpreting this statistic for each of these explanatory variables. We provide 
these statistics as a descriptive measure of the explanatory power of the variable, and they 
are not intended to be used as a measure of the size or “importance” of each variable. The 
betas are provided as a measure of importance. 
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Data Elements 
Achievement Gains 

We aggregated to the school level the vocational student-level achievement data for 
1996 and 1998. The level of analysis then became the school. We then used a measure of 
regressed change for our outcome variable. That is, we regressed the mean achievement in 
1998 on achievement in 1996, creating a posttest score that was regression-adjusted. We 
did this for each subject area: mathematics, science, and reading. Thus, the outcome 
measures of “change” were uncorrelated with the pretest scores and avoided many of the 
problems with simple change scores . 5 The model used in this process is shown below: 

Equation 1 

A 

Y = Bq + B test 9(iX test 96 

A 

Y = 1998 test score 
Control Variables 

One of the threats to the validity of this study was that within a school there might 
have been changes in the composition of test takers from 1996 to 1998. Differences in a 
school’s mean test score might be due to large differences in the composition of vocational 
completers from one year to the next and not to changes in program practices. We 
therefore controlled for changes in the racial-ethnic composition and socioeconomic status 
of the test takers by including variables measuring racial-ethnic composition, mean level of 
students’ fathers’ education, and mean level of students’ mothers’ education for both 1996 
and 1998. The model used is shown below: 

Equation 2 

Y = Bo + B tes t 96 ^test 96 + BjXj + -# 2-^2 + B 3 X 3 + B 4 X 4 + B 5 X 5 + B^Xfi 

X] = Percentage of school’s minority students in 1998 
X 2 = Percentage of school’s minority students in 1996 
X 3 = School mean father’s education level in 1998 
X 4 = School mean father’s education level in 1996 

5 For a discussion of regressed-adjusted change scores, see Burr and Nesselroade, 1990; and Cohen and 
Cohen, 1983. 
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X 5 = School mean mother’s education level in 1998 
X <3 = School mean mother’s education level in 1996 

Program Variables 

The HSTW initiative has a detailed list of key practices and objectives (see the 
Appendix). From reviewing these practices, we were able to establish six clusters of 
practices that grouped similar procedures and goals. These clusters were identified as 
(1) curriculum standards, (2) instructional goals, (3) academic/vocational integration, 
(4) guidance counseling, (5) teacher practices, and (6) work-based learning. 

As previously mentioned, however, the survey instruments used by the schools 
were not designed to examine specific changes over time in the implementation of the list 
of key practices nor the sets of related goals identified. The key practices are broad and 
complex; successive surveys were constructed to tap into and measure varying aspects of 
the broad practices of current interest to HSTW administrators, with only a subset of items 
remaining constant from year to year. Aspects of the program that were of interest to 
HSTW administrators in one year may not have been of critical interest the next; therefore, 
one of the challenges of this analysis was to identify common items from the 1996 and 
1998 surveys that measured the same aspects or clusters of key practices. Variables that 
measure most of the clusters of key practices within HSTW were identified; however, as 
will become apparent in the results section below, some clusters of practices were more 
fully operational than others were. 

The clusters of HSTW key practices that could be measured are described below. 
In the results section, the actual variables used to define these clusters are listed. The 
clusters of practices we examined were as follows: 

1 . Curriculum Standards 

The centerpiece of HSTW is raising the expectations of students by replacing the 
general track — a system that has traditionally allowed students to graduate from 
high school without completing a rigorous academic core — with a curriculum that 
blends the essential content of college preparatory science, mathematics, and 
language arts courses with challenging vocational/technical studies in grades 9 
through 12. Among other requirements, the HS7W-recommended curriculum calls 
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for at least three credits each in mathematics and science, with two credits in each 
subject from courses with content equal to that of college preparatory mathematics 
and science courses. The program of study should include science in the 11th or 
12th grade and mathematics in the senior year. 

2. Instructional Goals 

Meeting course requirements is just a part of the goals of the HSTW. Changing the 
content and instructional delivery practices of teachers is also a major goal. There is 
great emphasis on meeting the needs of all students through a set of challenging 
academic and vocational courses that actively engage each student in the learning 
process. 

3. Academic/Vocational Integration 

One of the main features of the reform of vocational education in the United States 
has been the attempt to integrate the study of academic subject matter into the 
vocational curriculum. In this way, it is hoped that students not served by traditional 
academic instructional methods can learn basic academic skills in an applied 
curriculum setting. .HSTW has long had this philosophy as a main component in its 
reform effort. 

4. Guidance Counseling 

Related to increased student expectations but seen as a different set of practices, is 
the guidance and counseling of students. Whether conducted predominantly by 
guidance counselors or shared with the teaching staff, the function of career 
guidance and counseling — and encouragement to enroll in challenging courses — is 
a responsibility taken seriously in schools where improvement has occurred. 

5. Teacher Practices 

One goal of the HSTW initiative is to increase access to academic studies that teach 
the essential concepts from the college preparatory curriculum through functional 
and applied strategies that enable students to see the relationship between course 
content and their future. 

6. Work-Based Learning 

The HSTW initiative supports the philosophy that many students learn more 
effectively within a “real world” context — that is, within a “structured system of 
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work-based and school-based learning” that involves schools working with 
employers within the local community to provide a career/employment context for 
the students’ academic and vocational coursework. 

Table 1 shows the measures of central tendency, variances, and the range for each 
set of variables that were used to represent the six clusters of practices. The first column of 
Table 1 shows the mean for each variable in 1996, while column 2 shows the within- 
school variance for each variable, and column 3 shows the between-school variances for 
each variable. The between-school variance for most of these variables was approximately 
10 percent of total variance. 6 This indicates that the observed differences between the 
schools on these variables were not due to random error. 

Column 4 shows the mean difference between each variable in 1996 and the same 
variable in 1998. Table 1 also includes the variances and ranges for the change variable. We 
have also provided in Table 1 the measurement scale used for each variable or set of 
variables. 



6 For example, the between-school variance for the variable measuring the importance of the goal of 
social development was 0.039, accounting for 8.7 percent of total variance — 0.039/(0.039+0.417). 
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Table 1 

Univariate Statistics for the Variables Used in the Analysis 
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Change in talking to guidance counselor 1996 to 1998 2.01 0.642 0.054 0.02 -0.65 0.85 0.05 
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Table 1 (cont.) 
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RESULTS 

Gains in Achievement 

Table 2 displays the average gains between 1996 and 1998 in achievement in 
mathematics, reading, and science for the 424 schools in this study. The median gain 
ranged from 6 points in reading (about one-half of one standard deviation) to 12 points in 
mathematics (almost one full standard deviation). There was great variability in change 
though. Several schools had the measured achievement level of their vocational completers 
drop over the two years — some as much as 60 points. Some schools had impressive gains 
in achievement for their vocational completers — as much as 50 points in some instances; 
however, the reader should keep in mind that these mean scores are based on a set of 
vocational completers in each school and that each year represents a different cohort of 
vocational completers. As a result, some portion of the gains may be due to changes in the 
composition of vocational completers in a school from year to year. The mean number of 
vocational completers within schools was about 60 in 1998, with 15 as the minimum 
number and 261 as the maximum; however, equally impressive is the overall average 
gains by this set of schools. For example, the mean gain for all schools in mathematics 
was 13 points. Since the assessment is scaled to have a mean of 250 and a standard 
deviation of 50, this represents an average gain of about V 4 of one standard deviation. 



Table 2 

Change in Science, Mathematics, and Reading Test Scores: 1996 to 1998 





N 


Mean 


Median 


Std. Deviation 


Minimum 


Maximum 


Science 


424 


94 


9.5 


11.2 


-30.9 


39.7 


Mathematics 


424 


13.1 


12.3 


13.3 


-39.4 


54.8 


Reading 


424 


4.4 


5.6 


14.8 


-63.4 


49.2 



Figure 1 plots the residualized change scores for mathematics, science, and reading 
for the schools in the study. These scores are the deviations of the predicted 1998 school 

7 

means based on the 1996 school means derived from Equation 1 provided earlier. 



7 For those unfamiliar with box plots, the boundaries of the box are the 25 th and 75 th percentiles, while 
the line in the middle of the box is the 50 th percentile (the median). The symbol “O” in the plots 
represents “outliers” which are 1.5 inter-quartile ranges (IRQs) from the end of the box. (The IRQ is 
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Figure 1 

Residualized Change Scores for Science, Mathematics, and Reading 
Achievement: 1998 Scores Regressed on 1996 Scores 




Science Mathematics Reading 



As with the simple change scores above, schools showed a great deal of variation 
in their gains in achievement of their vocational completers as measured by deviations in 
1998 from their predicted scores based on their scores in 1996. Particularly striking were 
the schools that had large drops in achievement levels. Further investigation revealed that 
one school in particular (the one represented by the in the plots above) had scored 
much lower on all three of the assessment areas — dropping from 40 to 60 points in each 
assessment. This school also did not have an unusually low number of vocational 
completers, testing 36 in one year and 46 in the other. 



the length of the box.) The symbol “*” represents extreme values and is more than three IRQs from 
the end of the box. 
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Control Variables 

One explanation for this precipitous drop in achievement for some schools was that 
changes had occurred between 1996 and 1998 in the composition of the vocational 
completers in these schools. We therefore introduced a set of control variables into the 
model that attempted to measure changes in the social composition of the schools. This 
model is shown in Equation 2 mentioned previously. Table 3 shows the results from this 
analysis. Indeed, changes in the racial-ethnic composition and socioeconomic class of the 
schools’ vocational completers seemed to be particularly useful predictors — explaining 
from 11 to 15 percent of the variance in 1998 residualized test scores. Figure 2 plots the 
residualized change scores for science, mathematics, and reading with the demographic 
central variables added to the model. 

When these residuals were analyzed, the one outlier school stood out even more. 
After examining this school’s data more carefully, it was decided that while certainly an 
outlier in the statistical sense, other data from this school suggested that the drop in 
achievement levels was not just an error of some sort, but real. Furthermore, after running 
a few equations with and without this one case in the sample, we found few, if any 
substantive differences in the results and decided to include it in the rest of the analysis. 
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Table 3 

Regression Results of 1996 Mean Test Score and Control Variables on 1998 Mean Test Score 
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Figure 2 

Residualized Change Scores for Science, Mathematics, 
and Reading Achievement: 1998 Scores Regressed on 1996 Scores 

and Demographic Variables 




Science 



Mathematics 



Reading 



Program Variables 



Curriculum Standards 

One of the more easily measured key practices is whether students in participating 
sites are completing the HSTW recommended curriculum. Among other requirements, the 
/fS7W-recommended curriculum calls for at least three credits each in mathematics and 
science, with two credits in each subject from courses with content equal to that of college 
preparatory mathematics and science courses. On average, there was a large increase in the 
percentage of students completing the SREB recommended coursework in science and 
mathematics. Figure 3 and Table 1 shown previously depict the distribution of schools on 
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these two variables. The mean change was about +17 percentage points for both science 
and mathematics (see Figure 3 and Table 1). Not all changes were positive, however. A 
small percentage of schools (11 percent) had decreased by more than 10 percentage points 
the proportion of students completing the recommended curriculum. 



Figure 3 

Univariate Distribution of Changes in the Percentage of Students 
Completing the Recommended Curriculum in Science and Mathematics, 

by School: 1996 to 1998 




Change science Change mathematics 



Multivariate Analysis 

Increases in the proportion of students meeting HSTW s curriculum standards had 
a large impact on achievement gains. Table 4 displays the individual unstandardized 
regression coefficient in, the beta in, and the partial correlation coefficients for the two 
variables. Adding these variables into the equation as a set resulted in a significant 
increment in R over the control model for each subject area. They accounted for 4% more 
variance in science achievement, 8% in mathematics, and 5% in reading. 

Examining individual variables, a 1 -percentage-point change in the percentage of 
students completing the recommended mathematics curriculum was associated with a 1- to 
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2-point increase in science, mathematics, and reading test scores (1.16, 1.62, and 1.26 
points respectively). For example, let’s say that school A increased by 10 percent the 
percentage of students completing the recommended curriculum. One would then predict 
from the data that school A would also increase its average test score by 1 1 points — about 
one-half of one standard deviation. 



Table 4 

Regression Results of Changes in the Percentage of Students Completing 
Recommended Curriculum on Change in 1998 School Test Score Means, 
Controlling for Changes in Demographic Characteristics 



Science 


B 1 


Beta 2 


Partial 

Correlation 3 


R 2 added to 
control model 


Change in % completing math curriculum 


1.16 


0.189** 


0.242 




Change in % completing science curriculum 


1.03 


0.184** 


0.237 


0.038** 


Mathematics 


B 1 


Beta 2 


Partial 

Correlation 3 


R 2 added to 
control model 


Change in % completing math curriculum 


1.62 


0.264** 


0.336 




Change in % completing science curriculum 


1.49 


0.267** 


0.343 


0.076** 


Reading 


B 1 


Beta 2 


Partial 

Correlation 3 


R 2 added to 
control model 


Change in % completing math curriculum 


1.26 


• 0.205** 


0.224 




Change in % completing science curriculum 


1.19 


0.213** 


0.234 


0.049** 


Note: ** p < 0.01, * p < 0.05, t p < 0.10 

1 The un standardized regression coefficient that would occur if that variable was entered into the model 

2 The standardized regression coefficient that would occur if that variable was entered into the model 

3 The correlation of the variable with the test score variable after removing the linear effect of the other variables 
in the control model 



Instructional Goals 

Figure 4 and Table 1 shown previously display the distribution of seven variables 
that represent the change in the importance of certain instructional goals. All of the 
variables in this set were based on an item in the 1996 and 1998 teacher questionnaire that 
asked the following question: 



O 

ERLC 



19 



31 



NCRVE, MDS-1295 



How important are the following goals in your school? 

The goals listed in the questionnaire that were common to 1996 and 1998 are as 
follows: 

• Help students in their social development by stressing the ability to get along with 
and understand others. 

• Help all high school students master the essential content taught in college 
preparatory language arts, mathematics, and science courses. 

• Help students make realistic plans for what they will do after graduation. 

• Help students pursue a program of high school studies that will enable them to 
achieve their plans. 

• Develop students’ abilities to solve problems and think critically. 

• Help students complete a program that prepares them for both employment and 
further learning. 

• Encourage students’ use of high-level academic content — language arts, 
mathematics, and science — in solving real-world problems. 

Response categories ranged from 1 ( very important ) to 4 ( not at all important). 

On average, there was very little change between 1996 and 1998 on teacher ratings 
of these goals (see Figure 4). Mean change for these variables ranged from 0.02 to 0.03 
points on the 4-point scale. One reason for this was that for many schools there was very 
little room for change. Mean rankings for these goals in 1996 were between very important 
and important (1.58 to 1.85). Almost all schools’ teachers ranked these goals as important; 
even schools that had relatively low rankings for these goals gave them ranks of between 
important and not too important. 
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Figure 4 

Univariate Distribution of Changes in the Importance Teachers 
Placed on Certain Instructional Goals, by School: 1996 to 1998 




Multivariate Analysis 

Not surprisingly, given their low variances, these variables as a set did not add 
appreciably to the overall predictive power of the control equation. Taken together they 
added only 1 to 2% to the R 2 of the model (see Table 5). 

Individually, these variables also did not add much to the prediction of gains in 
achievement. The one exception was the goal of helping students pursue a program of high 
school studies that will enable them to achieve their plans. A 1-unit change in the average 
ranking of this goal (e.g., from not so important to important ) was associated with a 4- to 

g 

5-point gain in math and reading achievement (at the a=0.05 level). 



8 Change in this goal was associated with science achievement at the a=0.10 level. 
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Table 5 

Regression Results of Changes in the Importance of Instructional Goals 
on Change in 1998 School Test Score Means, Controlling for Changes 

in Demographic Characteristics 





B 1 




Partial 


R^ added to 


Science 


Beta 2 


Correlation^ 


control model 


Change in Importance of Goal 










Social development 


- 0.72 


- 0.01 


- 0.02 




Master essential content 


0.67 


0.01 


0.02 




Realistic plans 


4.63 


0 . 08 * 


0.11 




High school program to achieve their plans 


4.10 


0 . 061 * 


0.09 




Develop problem-solving and critical thinking 


0.39 


0.01 


0.01 




Prepare all students for further learning 


2.50 


0.04 


0.05 




High-level academics in real-world tasks and problems 


4.81 


0 . 08 * 


0.10 


0.02 




B 1 




Partial 


R^ added to 


Mathematics 


Beta 2 


Correlation^ 


control model 


Change in Importance of Goal 










Social development 


- 0.84 


- 0.01 


- 0.02 




Master essential content 


- 0.12 


0.00 


0.00 




Realistic plans 


3.02 


0.05 


0.07 




High school program to achieve their plans 


4.61 


0 . 07 * 


0.10 




Develop problem-solving and critical thinking 


0.01 


0.00 


0.00 




Prepare all students for further learning 


2.54 


0.04 


0.06 




High-level academics in real-world tasks and problems 


1.92 


0.03 


0.04 


0.01 








Partial 


R^ added to 


Reading 


B 1 


Beta^ 


Correlation^ 


control model 


Change in Importance of Goal 










Social development 


- 1.63 


- 0.03 


- 0.03 




Master essential content 


0.22 


0.00 


0.00 




Realistic plans 


3.93 


0.07 


0.08 




High school program to achieve their plans 


5.46 


0 . 09 * 


0.10 




Develop problem-solving and critical thinking 


1.76 


0.03 


0.03 




Prepare all students for further learning 


4.08 


0.07 


0.08 




High-level academics in real-world tasks and problems 


5.12 


0 . 081 * 


0.09 


0.02 


Note: ** p < 0 . 01 , * p < 0 . 05 , t p < 0.10 



1 The unstan dar dized regression coefficient that would occur if that variable was entered into the model 

2 The s tandar dized regression coefficient that would occur if that variable was entered into the model 

3 The correlation of the variable with the test score variable after removing the linear effect of the other variables in the control 

model 
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Academic/Vocational Integration 

Three variables comprised the set of variables used to represent change in how 
much the school integrated academic instruction with the vocational curriculum. All three 
were based on items on the 1996 and 1998 student questionnaire. They were as follows: 

Did you feel your academic and vocational teachers were working together 
to improve your skills in the following areas? 

• Reading 

• Writing 

• Math 

Students were asked to answer either yes (coded here as 1) or no (coded here as 0). 

Figure 5 displays the distribution of these three variables. Table 1 (shown ' 
previously) gives the descriptive statistics for these variables. The baseline data in 1996 
shows that on average these schools were engaged in a high degree of academic/vocational 
integration. On average, in 1996, schools had about 71 to 75% of their students responding 
that their academic and vocational teachers worked together to improve their academic 
skills. Nevertheless, there appears to be an overall increase in the amount of academic and 
vocational integration between 1996 and 1998. Mean change from 1996 to 1998 was 5% 
for teachers working together on reading skills, 6% on writing skills, and 2% for math 
skills. 
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Figure 5 

Univariate Distribution of Changes 
in Academic and Vocational Integration, by School: 1996 to 1998 
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Multivariate Analysis 

The introduction of this set of variables did not result in a significant increment in 
R for science, mathematics, or reading (see Table 6); however, change in the proportion of 
students saying that their teachers worked together to improve their math skills added 
significantly to the prediction of mathematics achievement. Change in teachers’ 
coordination to improve writing and reading skills added significantly to the prediction of 
gains in reading achievement. 
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Table 6 

Regression Results of Changes in the Amount of Time 
Vocational and Academic Teachers Worked Together on Change 
in 1998 Mean Test Scores Controlling for Demographic Characteristics 









Partial 


R^ added to 


Science 


B 1 


Beta 2 


Correlation^ 


control model 


Change in Teachers Working Together To Improve . . . 










Reading skills 


0.23 


0.022 


0.030 




Writing skills 


0.40 


0.036 


0.048 




Math skills 


0.56 


0.046 


0.063 


0.003 








Partial 


R^ added to 


Mathematics 


B 1 


Beta 2 


Correlation^ 


control model 


Change in Teachers Working Together To Improve . . . 










Reading skills 


0.39 


0.04 


0.05 




Writing skills 


0.48 


0.04 


0.06 




Math skills 


1.20 


0.10** 


0.13 


0.010 








Partial 


R^ added to 


Reading 


B 1 


Beta^ 


Correlation^ 


control model 


Change in Teachers Working Together To Improve . . . 










Reading skills 


1.21 


0.12** 


0.14 




Writing skills 


1.31 


0.12** 


0.14 




Math skills 


0.84 


0.07 


0.08 


0.017 


Note: ** p < 0.01, * p < 0.05, t p < 0.10 










1 The unstandardized regression coefficient that would occur if that variable was entered into the model 

2 The standardized regression coefficient that would occur if that variable was entered into the model 




3 The correlation of the variable with the test score variable after removing the linear effect of the other variables m the control 


model 











In addition to statistical significance, the effect sizes of these variables were 
relatively large. In fact, they were almost as large as the effect sizes of changes in the 
percentage of students meeting or exceeding the HSTW curriculum standards. A 1% 
increase in the proportion of a school’s students reporting that their academic and 
vocational teachers worked together to help improve their math skills resulted in a predicted 
1.2-point increase in mathematics achievement. Likewise, a 1 -point increase in the 
proportion of a school’s students reporting teachers working together to improve reading 
and writing resulted in an increase of about 1.2 to 1.3% in reading achievement. This 
represents more than a one-to-one correspondence. 
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Guidance Counseling of Students 

Two variables were used in this set. They were based on student responses to the 
following items in 1996 and 1998: 

How much have you talked with the following people about planning your 
school program? 

Choices included the following: 

• A guidance counselor 

• Teachers 

Response categories were not at all (coded 1), somewhat (coded 2), and a great deal 
(coded 3). Figure 6 shows the univariate distribution of these two variables, while Table 1 
(shown previously) displays the descriptive statistics. 

On average, students within the HSTW schools had not talked very much with 
either their counselors or their teachers about their high school program. In 1996, the 
average student within a school had spoken with their counselor or teacher about their high 
school program only somewhat (mean=2.01 and 1.91 respectively). There also had been 
little or no change between 1996 and 1998 in the average amount of time students spent 
talking with their counselor or teacher (mean=0.02 and 0.05 respectively). Despite the 
small average difference between 1996 and 1998, some schools substantially increased the 
amount of time their students spent talking with their counselors and teachers, while others 
considerably decreased the amount of time their vocational completers spoke with these 
school personnel. 



ERIC 



26 



38 



NCRVE, MDS-1295 



Figure 6 

Univariate Distribution of Changes in the Proportion of Students 
Talking to Their Counselor or Teacher About Their High School Program, 

by School: 1996 to 1998 

1.0 



.5 



0.0 



-.5 



- 1.0 

Guidance counselor Teacher 




Multivariate Analysis 

There was a strong association of changes in the amount that students talked with a 
teacher or school counselor and changes in school achievement levels in science, 
mathematics, and reading. Controlling for demographic characteristics, those schools that 
increased the amount of time that students talked to teachers and counselors about their 
high school program increased their achievement rates; those that decreased this time had 
declines in their average achievement levels. The increment to the proportion of explained 
variance for the set of variables ranged from 3 to 4% (see Table 7). The effect sizes for the 
two variables were also relatively large, with standardized regression coefficients (Beta 
in’s) ranging from 0.12 to 0.18. The unstandardized regression coefficients suggest that a 
1-point change on this 3-point scale (e.g., from not at all to somewhat ) was associated with 
an 8- to 1 1 -point change in academic achievement. 
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Table 7 

Regression Results of Changes in the Amount of Time 
Students Spent Talking with Counselors and Teachers About 
Their High School Plans, Controlling for Demographic Characteristics 





B 1 


Beta ^ 


Partial 


R 2 added to 


Science 


Correlation^ 


control model 


Change in Talking About High School Plans with . . . 










Counselor 


7.85 


0.141** 


0.191 




Teacher 


8.65 


0.120** 


0.162 


0.025* 




B 1 




Partial 


R 2 added to 


Mathematics 


Beta 2 


Correlation 2 


control model 


Change in Talking About High School Plans with . . . 










Counselor 


8.45 


0.15** 


0.21 




Teacher 


8.16 


0.11** 


0.15 


0.026* 




B 1 




Partial 


R 2 added to 


Reading 


Beta 2 


Correlation 2 


control model 



Change in Talking About High School Plans with . . . 

Counselor 9.93 0.18** 0.21 

Teacher 11.66 0.16** 0.19 



0,04* 

Note: ** p < 0.01, * p < 0.05, t p < 0.10 

1 The unstandardized regression coefficient that would occur if that variable was entered into the model 

2 The standardized regression coefficient that would occur if that variable was entered into the model 

3 The correlation of the variable with the test score variable after removing the linear effect of the other variables in the control 

model 



Teacher Practices 

Eight variables measuring teacher practice were common to the 1996 and 1998 
teacher questionnaires. They were based on the following item: 

To what extent has your emphasis on the following practices changed since 
your school became an SREB High Schools That Work site? 

The practices included the following: 

• Engaging students in learning activities that involve academic content 

• Using manipulatives and hands-on experiments or projects to make content more 
concrete 

• Students doing joint assignments in which they work with an academic and a 

vocational teacher 
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• Amount of homework assigned and reviewed 

• Having students write to clarify and communicate their ideas 

• Asking students to use mathematics to solve challenging real-world problems 

• Amount of time students spend on assigned reading 

• Students taking greater responsibility for their learning 

Response categories ranged from 1 ( much less ) to 5 ( much more), with 3 
representing no change. Figure 7 displays the univariate distribution of these variables 
while Table 1 (shown previously) supplies descriptive statistics for these variables. 



Figure 7 

Univariate Distribution of Changes in Teacher Practices, 
by School: 1996 to 1998 
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The interpretation of these variables is a little tricky since teachers were asked in 
1996 and 1998 how much their schools had changed practice, and then, we measured the 
difference in teachers’ assessments between each period. We are, therefore, measuring the 
change in how much teachers think that change has occurred. Given this caution, on 
average, there was relatively little change within schools in teacher practices. On the 5-point 
scale used for these variables, average change was less than one-quarter of one point. 
Furthermore, even the maximum change was only about 1 Vi points. 



Multivariate Analysis 

Given this lack of variance, it is not surprising that changes in teacher practice did 
not have much impact on the prediction of school achievement levels. As a set, these 
variables did not add significantly to the R of the model that contained only the control 
variables (see Table 8). The only variable within this set that seemed to have some limited 
impact was change in having students do joint projects; however, the statistical confidence 
of this finding was only at the a 
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Table 8 

Regression Results of Changes in the Teacher Practices on Change 
in Mean Test Scores, Controlling for Changes in Demographics 









Partial 


added to 


Science 


B 1 


Beta 2 


Correlation^ 


control model 


Change in Teacher Practice of . . . 










Engaging students in learning that involves academics 


2.60 


0.05 


0.07 




Using manipulatives and hands-on projects to make 






-0.02 




content concrete 


-0.87 


-0.02 




Having students do joint assignments 


3.81 


0.08* 


0.11 




Amount of homework assigned and reviewed 


3.20 


0.061* 


0.08 




Amount of time students write 


2.14 


0.04 


0.05 




Amount of time students use math to solve real-world 










problems 


2.32 


0.04 


0.06 




Amount of time students spend reading 


1.09 


0.02 


0.03 




Getting students to take greater responsibility 


1.33 


0.03 


0.04 










0.02 








Partial 


added to 


Mathematics 


B 1 


Beta^ 


Correlation^ 


control model 


Change in Teacher Practice of .. . 










Engaging students in learning that involves academics 


1.15 


0.02 


0.03 




Using manipulatives and hands-on projects to make 






-0.03 




content concrete 


-0.95 


-0.02 


, 


Having students do joint assignments 


2.68 


0.061* 


0.08 




Amount of homework assigned and reviewed 


1.60 


0.03 


0.04 




Amount of time students write 


2.06 


0.04 


0.05 




Amount of time students use math to solve real-world 










problems 


1.76 


0.03 


0.05 




Amount of time students spend reading 


0.41 


0.01 


0.01 




Getting students to take greater responsibility 


1.38 


0.03 


0.04 










0.01 








Partial 


R^ added to 


Reading 


B 1 


Beta^ 


Correlation^ 


control model 


Change in Teacher Practice of. . . 










Engaging students in learning that involves academics 


3.26 


0.07 


0.08 




Using manipulatives and hands-on projects to make 






0.01 




content concrete 


0.29 


0.01 




Having students do joint assignments 


3.65 


0.081* 


0.09 




Amount of homework assigned and reviewed 


3.96 


0.081* 


0.09 




Amount of time students write 


3.03 


0.05 


0.06 




Amount of time students use math to solve real-world 










problems 


4.50 


0.09* 


0.10 




Amount of time students spend reading 


2.01 


0.04 


0.05 




Getting students to take greater responsibility 


3.48 


0.07 


0.08 










0.02 


Note: ** p < 0.01, * p < 0.05, t P < 0.10 

1 The unstandardized regression coefficient that would occur if that variable was entered into the model 

2 The standardized regression coefficient that would occur if that variable was entered into the model 

3 The correlation of the variable with the test score variable after removing the linear effect of the other variables in the control model 
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Work-Based Learning 

Only two variables were available to be used to represent the key practice of work- 
based learning. These were derived from items on the student questionnaire. 

What best describes the amount of emphasis your vocational teachers 
placed on .. . having an expert outside the school evaluate [your] work, 
products, or accomplishments? 

Response categories were never required (coded as 1), required once or twice a year 
(coded as 2), required monthly or several times a year (coded as 3), and required daily or 
weekly (coded as 4). 

I participated in a work-based internship for which I completed a written 
and/or oral report 

Response categories were yes (coded as 1) and no (coded as 0). 

Figure 8 displays the univariate distribution of change in vocational teachers using 
outside experts while Figure 9 displays the same information for work-based internships. 
Table 1 presents descriptive statistics for these variables. 




32 



44 



NCRVE, MDS-1295 



Figure 8 

Univariate Distribution of Changes in Use of Outside Experts 
To Review Vocational Work and Projects, by School: 1996 to 1998 
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Figure 9 

Univariate Distribution of Changes in Percentage of Vocational Completers 
Participating in Work-Based Internships, by School: 1996 to 1998 
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A large percentage of vocational completers participated in a work based-intemship 
in 1996 (about 31%). On average, within schools this percentage increased about 4 
percentage points between 1996 and 1998 — to approximately 35% in 1998. Having 
vocational work or projects reviewed by outside experts was a more rare event, relatively 
speaking. On average in 1996, schools had completers who reported either never having 
outside experts review their work or having this requirement only once or twice a year 
(mean=1.87 on the 4-point scale). There was little change in this mean ranking between 
1996 and 1998 with a mean decrease of one-tenth of one point. 

Multivariate Analysis 

These two variables did not significantly add to the explanatory power of the model 
for achievement in science, mathematics, or reading (see Table 9). However, for 
mathematics, the beta in for each variable was statistically significant, and the effect was 
negative on achievement. Holding the demographic variables constant, increases in the 
proportion of work-based internships, and increases in the time vocational teachers spent 
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using outside experts to evaluate student work were associated with decreases in academic 
achievement. 



Complete Models 

While the modeling approach used above has been simple, this approach has led us 
to more interpretable results; however, we also feel that it is worthwhile to assess the total 
impact on student achievement of the SREB key practices, as measured in this analysis. 
Therefore, in this section, the results of regressing science, mathematics, and reading 
achievement on the full array of variables used to represent the HSTW key practices are 
presented. 

Tables 10, 11, and 12 present the control model (model 1) and the model with the 
full array of variables (model 2) for science, mathematics, and reading achievement, 
respectively. The HSTW key practices as measured in this analysis accounted for 13% of 
the variance between schools in their science and mathematics achievement scores and 9% 
of the variance in reading achievement scores. 

These are still fairly simple models in which all the relationships are hypothesized 
to be linear, and we do not examine any interactions. Furthermore, the number of 
interrelated variables in this model makes straightforward interpretations of the data 
difficult. For example, in the full model, there is a consistent and strong negative 
relationship between the use of manipulatives and changes in science, mathematics, and 
reading achievement; however, these same variables had virtually no relationship with 
gains in achievement in our simple models presented above (although the sign of the 
effects were the same). While it is always possible to find some post hoc explanation for 
this kind of result, it is difficult to imagine what theory would predict such an outcome. 
Given the complexity of the interrelationships between these variables and a lack of theory 
on how they would interact, we feel that the simple models presented in this study are 
much more useful. 
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Table 9 

Regression Results of Changes in the Work-Based Learning 
Practices Change in 1998 Mean Test Scores, 
Controlling for Changes in Demographics 





B 1 




Partial 


R 2 added to 


Science 


Beta 2 


Correlation^ 


control model 


Vocational teacher emphasis on having outside 










expert evaluate work 


-2.39 


-0.050 


-0.068 




Participated in work-based internship 


-0.09 


-0.085* 


-0.114 


0.008 








Partial 


R 2 added to 


Mathematics 


B 1 


Beta 2 


Correlation^ 


control model 


Vocational teacher emphasis on having outside 










expert evaluate work 


-5.16 


-0.11** 


-0.15 




Participated in work-based internship 


-0.13 


-0.12** 


-0.17 


0.02 








Partial 


R 2 added to 


Reading 


B 1 


Beta 2 


Correlation^ 


control model 


Vocational teacher emphasis on having outside 










expert evaluate work 


-2.47 


-0.05 


-0.06 




Participated in work-based internship 


-0.09 


-0.07 


-0.07 


0.01 



Note: ** p < 0.01, * p < 0.05, t P < 0.10 

1 The unstandardized regression coefficient that would occur if that variable was entered into the model 

2 The standardized regression coefficient that would occur if that variable was entered into the model 

3 The correlation of the variable with the test score variable after removing the linear effect of the other variables in the control 

model 
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Table 10 

Results of Full Regression Model of Key Practices Variables 
on Change in 1998 Science Test Scores 



(Constant) 

1996 test 

Percent minority 1998 
Percent minority 1996 
Mean fathers’ educational level 1998 
Mean fathers’ educational level 1996 
Mean mothers’ educational level 1998 
Mean mothers’ educational level 1996 

Change in % completing math curriculum 
Change in % completing science curriculum 

Change in importance of goal 
Social development 
Master essential content 
Realistic plans 

High school program to achieve their plans 
Develop problem-solving and critical thinking 
Prepare all students for further learning 
High-level academics in real-world tasks and problems 

Change in teachers working together to improve . . . 

Reading skills 
Writing skills 
Math skills 

Change in talking about high school plans with . . . 

Counselor 

Teacher 

Change in teacher practice of. . , 

Engaging students in learning that involves academics 
Using manipulatives and hands-on projects to make content 
concrete 

Having students do joint assignments 
Amount of homework assigned and reviewed 
Amount of time students write 
Amount of time students use math to solve real-world 
problems 

Amount of time students spend reading 
Getting students to take greater responsibility 

Vocational teacher emphasis on having outside expert 
evaluate work 

Participated in work-based internship 
Total R 2 

Incremental R 2 

Note ; ** p < 0.01, * p < 0.05, t p < 010 



Model 1 


Model 2 


B 


Beta 


B 


Beta 


151.58 




129.17 




0.38 


0.33** 


0.46 


0.40** 


-24.91 


-0.53** 


-28.56 


-0.60** 


18.47 


0.38** 


23.81 


0.49** 


10.25 


0.25** 


7.39 


0.18* 


0.92 


0.02 


3.79 


0.09 


2.91 


0.06 


0.62 


0.01 


-2.24 


-0.05 


-1.42 


-0.03 






7.56 


0.11* 






6.68 


0.11* 






-6.55 


-0.091 






-4.38 


-0.06 






-2.36 


-0.03 






7.38 


0.10 






-5.36 


-0.07 






4.08 


0.06 






8.16 


0.1 it 






8.15 


0.07 






5.18 


0.04 






-2.97 


-0.02 






6.58 


0.11* 






9.04 


0.11* 






-1.33 


-0.02 






-8.99 


-0.15** 






1.26 


0.02 






2.07 


0.03 






-2.29 


-0.03 






5.42 


0.08 






-0.67 


-0.01 






7.16 


0.12* 






-2.83 


-0.05 






-0.09 


-0.08f 




0.271** 




0.398** 




0.271** 




0.127** 



NCRVE, MDS-1295 



Table 11 

Results of Full Regression Model of Key Practices Variables 
on Change in 1998 Math Test Scores 



(Constant) 

1996 test 

Percent minority 1998 
Percent minority 1996 
Mean fathers’ educational level 1998 
Mean fathers’ educational level 1996 
Mean mothers’ educational level 1998 
Mean mothers’ educational level 1996 

Change in % completing math curriculum 
Change in % completing science curriculum 

Change in importance of goal 
Social development 
Master essential content 
Realistic plans 

High school program to achieve their plans 
Develop problem-solving and critical thinking 
Prepare all students for further learning 
High-level academics in real-world tasks and problems 

Change in teachers working together to improve . . . 

Reading skills 
Writing skills 
Math skills 

Change in talking about high school plans with . . . 

Counselor 

Teacher 

Change in teacher practice of . . . 

Engaging students in learning that involves academics 
Using manipulatives and hands-on projects to make content 
concrete 

Having students do joint assignments 
Amount of homework assigned and reviewed 
Amount of time students write 
Amount of time students use math to solve real-world 
problems 

Amount of time students spend reading 
Getting students to take greater responsibility 

Vocational teacher emphasis on having outside expert 
evaluate work 

Participated in work-based internship 
Total R 2 

Incremental R 2 

Note: ** p < 0.01, * p < 0.05, f p < 0.10 



Model 1 


Model 2 


B 


Beta 


B 


Beta 


178.99 




152.53 




0.37 


0.43** 


0.46 


0.54** 


-20.77 


-0.49** 


-21.72 


-0.51** 


11.53 


0.27* 


14.29 


0.33** 


12.83 


0.34** 


8.72 


0.23** 


-0.14 


0.00 


1.87 


0.05 


4.57 


0.10 


2.61 


0.06 


-6.13 


-0.14| 


-3.78 


-0.09 



9.32 0.15** 

7.73 0.14** 



-2.41 -0.04 

-2.43 -0.04 

-0.82 -0.01 

6.33 0.09| 

-5.83 -0.09| 

3.16 0.05 

2.77 0.04 

-1.38 -0.01 

1.39 0.01 

7.78 0.06| 

5.52 0.10** 

5.48 0.08* 

-0.52 -0.01 

-7.30 -0.14** 

1.34 0.03 

0.65 0.01 

0.05 0.00 

3.10 0.05 

-0.80 -0.01 

3.47 0.06 

-3.82 -0.08* 

- 0.10 - 0 . 10 ** 

0.483** 0.614** 

0.483** 0.131** 
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Table 12 

Results of Full Regression Model of Key Practices Variables 
on Change in 1998 Reading Test Scores 



(Constant) 

1996 test 

Percent minority 1998 
Percent minority 1996 
Mean fathers’ educational level 1998 
Mean fathers’ educational level 1996 
Mean mothers’ educational level 1998 
Mean mothers’ educational level 1996 

Change in % completing math curriculum 
Change in % completing science curriculum 

Change in importance of goal 
Social development 
Master essential content 
Realistic plans 

High school program to achieve their plans 
Develop problem-solving and critical thinking 
Prepare all students for further learning 
High level academics in real world tasks and problems 

Change in teachers working together to improve . . . 

Reading skills 
Writing skills 
Math skills 

Change in talking about high school plans with . . . 

Counselor 

Teacher 

Change in teacher practice of. . . 

Engaging students in learning that involves academics 
Using manipulates and hands-on projects to make content 
concrete 

Having students do joint assignments 
Amount of homework assigned and reviewed 
Amount of time students write 
Amount of time students use math to solve real-world 
problems 

Amount of time students spend reading 
Getting students to take greater responsibility 

Vocational teacher emphasis on having outside expert 
evaluate work 

Participated in work-based internship 
Total R 2 

Incremental R 2 

Note: ** p < 0,01, * p < 0.05, t P < 010 



Model 1 


Model 2 


B 


Beta 


B 


Beta 


154.98 




132.50 




0.44 


0.38** 


0.51 


0.44** 


-25.36 


-0.60** 


-27.27 


-0.64** 


13.41 


0.31** 


16.70 


0.39** 


9.10 


0.24** 


6.29 


0.17* 


1.23 


0.03 


2.98 


0.08 


5.57 


0.13t 


4.17 


0.10 


-4.10 


-0.11 


-2.57 


-0.06 






7.36 


0.12** 






4.40 


0.08t 






-2.89 


-0.04 






-2.80 


-0.04 






3.33 


0.05 






2.11 


0.03 






-7.36 


-0.11* 






1.52 


0.02 






8.86 


0.13* 






-2.43 


-0.02 






1.47 


0.01 






1.52 


0.01 






4.50 


0.08* 






6.30 


0.09* 



1.39 0.03 

-8.16 -0.16** 

2.73 0.06 

1.73 0.03 

-0.70 -0.01 

2.04 0.04 

-0.43 -0.01 

2.14 0.04 

-1.66 -0.03 

-0.08 -0.08* 

0.472** 0.562** 

0.472** 0.090** 
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However, it is reassuring that given all of these caveats that the main findings of 
our simple descriptive models hold when examined in the context of these more complete 
models. In these more complete models, an increase in the percentage of students meeting 
the recommended curriculum requirements and an increase in the amount of time students 
spent talking with counselors and teachers about their high school plans had a positive 
effect on science, mathematics, and reading achievement. The negative relationship 
between the variables used to measure work-based learning and achievement was also 
confirmed in these more elaborate models. 
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DISCUSSION 

In this analysis, we have examined several clusters of key practices that comprise 
the reform methods endorsed by the HSTW network and assessed their impact on 
academic achievement. Because of the large number of recommended reform methods and 
the large number of variables that could be used to represent these methods, we have tried 
to be particularly careful in using simple and direct analytical methods. We have chosen to 
examine simple models rather than over-complex explanatory models. We did this to 
educe some broad generalizations about key aspects of the HSTW effort without making 
claims to a degree of scientific rigor that was not practical nor warranted by the nature of 
the data that we used. 

In light of this, we would also like to reiterate our cautionary note about the nature 
of the data that we used in this analysis. These data were not collected for the purpose of an 
overall evaluation of the theoretical underpinnings of the HSTW reform effort. 
Furthermore, for many of the clusters of key practices within HSTW, we found only a 
limited number of variables common to the 1996 and 1998 surveys that we could use to 
operationalize the practice. The data are also not longitudinal, and this introduces additional 
concerns about cause and effect relationships. Furthermore, we would like to emphasize 
that we looked only at academic achievement. Many of the reforms we examine may be 
targeted at other outcomes (such as occupational outcomes) and not at academic 
achievement per se. 

Nevertheless, we did find evidence for the effectiveness of several key practices 
within the constellation of factors that comprise the HSTW reform effort. Specifically, we 
found that independent of schools’ demographic profile (or changes therein) the following 
factors are associated with gains in academic achievement: 

• An increase in the percentage of vocational completers within the school that 
complete a rigorous course of study in science and mathematics 

• An increase in the frequency with which vocational completers speak with their 
counselor or teacher about their high school program 
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We also found evidence for the effectiveness of integrating academic content into 
the vocational curriculum. Increases in the proportion of students within a school saying 
that their academic and vocational teachers were working together to improve their math 
skills were associated with gains for those schools in mathematics. Similarly, there was an 
increase in reading achievement in those schools for which there was an increase in the 
proportion of students who felt their academic and vocational teachers were working 
together to improve their reading and writing skills. There were also limited and tentative 
indications that increasing the proportion of students doing joint projects in which the 
student works with both a vocational and academic teacher improved a school’s academic 
achievement. 

Our findings also suggest that schools that are implementing work-based training 
or internships should look carefully at how they implement these programs. The results of 
this analysis suggest that these efforts may not pay off in terms of gains in achievement. In 
the schools that we examined here, there was a clear negative association of more students 
participating in these programs with lower achievement levels. 

In any analysis of schools, however, the direction of cause and effect can be 
difficult to determine. It may be that lower achieving students are more likely to enroll in 
work-based internships. Since the data we used here are not longitudinal, an increase within 
a school in the proportion of vocational students in work-based internships may reflect 
changes in the composition of their vocational students and not changes in policy. 
Therefore, schools that have more vocational completers participating in work-based 
internships in 1998 than in 1996 may have lower achievement levels due to these changes 
in the prior achievement levels of their vocational students and not due to changes in policy. 

Of course, the problem with the direction of cause and effect is present in the 
interpretation of those practices we found that we cautiously interpreted as signs of positive 
effects of the HSTW reform effort; however, we have tried to model changes in the 
composition of the schools’ vocational students with the control variables introduced into 
our models. While these controls are imperfect, the variables we used are traditionally 
correlated very highly with prior academic achievement. We also chose to examine the 
HSTW practices with simple models in the hope that this would minimize other 
confounding effects. 
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We chose to use data that had previously been collected by the schools within the 
HSTW network. These data were used by the schools for internal monitoring and were not 
specifically designed for evaluative purposes. As we mentioned earlier, these kinds of data 
are being collected at the local level at an ever-increasing frequency. While our primary 
goal of this study is to examine the correlates of success in the HSTW network, we also 
hope that our methods will stimulate others to think about how this kind of data could be 
used for research. 

Because of the limitations of the methods and data that we used, we were cautious 
about overinterpreting the results of the analysis; however, because the models were 
simple, we feel that they are all the more worthwhile to practitioners. Coupled with other 
quantitative and qualitative information, these results can be used by HSTW to focus their 
attention on particular aspects of their program. 

Sir Isaiah Berlin (1996) made the now famous distinction between understanding 
and knowledge — understanding involves insight into the world; knowledge involves only 
the accumulation of facts. We hope that the results presented in this paper will lead to 
policymakers and researchers building up a reservoir of relevant information that will lead 
to an understanding of how schools work and how to improve them. 
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APPENDIX 

HSTW KEY PRACTICES 

Binding the HSTW schools together is their willingness and effort to enact the 
following set of ten key practices, which aim to improve student performance by blending 
higher level academic studies and challenging career studies: 

1. Set high expectations, and have career-bound students meet them. 

2. Increase access to challenging vocational and technical studies, and emphasize the 
use of high-level academic skills in the context of the modem workplace and in 
preparation for continued learning. 

3. Increase access to academic studies that teach the essential concepts from the 
college preparatory curriculum through functional and applied strategies that enable 
students to see the relationship between course content and their future. 

4. Have students complete a challenging program of study with an upgraded academic 
core and a major. The academic core includes at least four years of college 
preparatory English and three years each of mathematics and science, with at least 
two years in each area equivalent in content to courses offered in the college 
preparatory program. The major includes at least four Carnegie units in a career or 
academic major and two Carnegie units in related technical core courses. 

5. Provide students with access to a structured system of work-based and school- 
based learning — secondary and postsecondary — collaboratively planned by 
educators, employers, and workers. 

6. Have an organizational structure and schedule that enables academic and vocational 
teachers to have the time to plan and provide integrated instruction. 

7. Have each student actively engaged in the learning process. 

8. Involve each student and his or her parent(s) in a career guidance and individualized 
advisement system. 
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9. Provide a structured system of extra help to enable career-bound students to 
successfully complete an accelerated program of study that includes high-level 
academic content and a major. 

10. Use student assessment and program evaluation data to continuously improve 
curriculum, instruction, school climate, organization, and management to advance 
student learning. 
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